{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9dad9e35",
   "metadata": {},
   "source": [
    "# 评价指标\n",
    "我们在初识自定义的时候就接触过了评价指标，我们需要用评估函数（Metrics）来评估模型的好坏，不同的训练任务通常会使用不同的评估函数，例如，对于二分类问题，常用的评价指标有precision（准确率）、recall（召回率）等，而对于多分类任务，可使用宏平均（Macro）和微平均（Micro）来评估。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5c4bd92",
   "metadata": {},
   "source": [
    "MindSpore提供了大部分常见的评估函数接口，我们也一样可以自定义评估函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18ee651f",
   "metadata": {},
   "source": [
    "## 一、自定义Metrics\n",
    "自定义Metrics需要继承`nn.Metric`父类，并重写其中的`clear方法、update方法和eval方法`。\n",
    "\n",
    "- `clear`：初始化相关的内部参数。\n",
    "- `update`：接收网络预测输出和标签，计算误差，每次step后并更新内部评估结果。\n",
    "- `eval`：计算最终评估结果，在没次epoch结束后计算最终的评估结果。\n",
    "\n",
    "下面我们以MAE算法为例：\n",
    "$$ MAE=\\frac{1}{n}\\sum_{i=1}^n\\lvert ypred_i - y_i \\rvert $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dc321e0f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:33:53.486151Z",
     "start_time": "2022-11-12T08:33:53.466170Z"
    }
   },
   "outputs": [],
   "source": [
    "import mindspore as ms\n",
    "import mindspore.nn as nn\n",
    "import numpy as np\n",
    "\n",
    "class MyMAE(nn.Metric):\n",
    "    \"\"\"自定义MAE\"\"\"\n",
    "    def __init__(self):\n",
    "        super(MyMAE, self).__init__()\n",
    "        self.clear() # 是clear在初始化的时候被调用\n",
    "    \n",
    "    def clear(self):\n",
    "        # 初始化需要用的变量\n",
    "        self.abs_error_sum = 0\n",
    "        self.samples_num = 0\n",
    "    \n",
    "    def update(self, *inputs):\n",
    "        # 在每完成一个step后更新参数\n",
    "        y_pred = inputs[0].asnumpy()\n",
    "        y = inputs[1].asnumpy()\n",
    "\n",
    "        abs_error = np.abs(y - y_pred)\n",
    "        self.abs_error_sum += abs_error.sum()\n",
    "        \n",
    "        self.samples_num += y.shape[0]\n",
    "        \n",
    "    def eval(self):\n",
    "        # 计算最终结果\n",
    "        return self.abs_error_sum / self.samples_num"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3405f1bb",
   "metadata": {},
   "source": [
    "测试MAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "22822a8a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:32:37.864871Z",
     "start_time": "2022-11-12T08:32:37.829197Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: 1.6666666666666667\n"
     ]
    }
   ],
   "source": [
    "y = ms.Tensor(np.array([1, 2, 3]), ms.float32)\n",
    "y_pred = ms.Tensor(np.array([2, 4, 5]), ms.float32)\n",
    "\n",
    "mae = MyMAE()\n",
    "mae.clear()\n",
    "mae.update(y_pred, y)\n",
    "error = mae.eval()\n",
    "print(\"Error:\", error)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "880dabf2",
   "metadata": {},
   "source": [
    "我们在`update方法`中使用的行参是`*inputs`，故我们可以多个输入，此时我们可以使用`set_indexes`方法来指定使用哪个输入来计算评估指标，需要注意的是，使用`set_indexes`方法，需要用装饰器`nn.rearrange_inputs`修饰`update`方法，否则使用set_indexes配置的输入不生效。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ae8d459",
   "metadata": {},
   "source": [
    "我们重写MyMAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7ac051e2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:43:24.726537Z",
     "start_time": "2022-11-12T08:43:24.712332Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyMAE(nn.Metric):\n",
    "    \"\"\"自定义MAE\"\"\"\n",
    "    def __init__(self):\n",
    "        super(MyMAE, self).__init__()\n",
    "        self.clear() # 是clear在初始化的时候被调用\n",
    "    \n",
    "    def clear(self):\n",
    "        # 初始化需要用的变量\n",
    "        self.abs_error_sum = 0\n",
    "        self.samples_num = 0\n",
    "        \n",
    "    @nn.rearrange_inputs\n",
    "    def update(self, *inputs):\n",
    "        # 在每完成一个step后更新参数\n",
    "        y_pred = inputs[0].asnumpy()\n",
    "        y = inputs[1].asnumpy()\n",
    "\n",
    "        abs_error = np.abs(y - y_pred)\n",
    "        self.abs_error_sum += abs_error.sum()\n",
    "        \n",
    "        self.samples_num += y.shape[0]\n",
    "        \n",
    "    def eval(self):\n",
    "        # 计算最终结果\n",
    "        return self.abs_error_sum / self.samples_num"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3097c81c",
   "metadata": {},
   "source": [
    "测试New MAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3bf81861",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:43:26.454231Z",
     "start_time": "2022-11-12T08:43:26.436325Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: 0.3333333333333333\n"
     ]
    }
   ],
   "source": [
    "# 增加z变量\n",
    "z = ms.Tensor(np.array([3, 4, 5]), ms.float32)\n",
    "\n",
    "mae = MyMAE().set_indexes([0, 2]) # 设置使用第一个入参和第三个入参计算MAE\n",
    "mae.clear()\n",
    "mae.update(y_pred, y, z)\n",
    "error = mae.eval()\n",
    "print(\"Error:\", error)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1ad4782",
   "metadata": {},
   "source": [
    "# 二、在Model接口中使用Metrics\n",
    "## 使用内置Metrics\n",
    "MindSpore内置的Metrics在`mindspore.nn`中,使用Model接口调用时，只需将指定其中的`metrics`参数即可。Metrics可以定义为一个字典类型，字典的key值为字符串类型，字典的value值为MindSpore内置的评价指标。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03ccce27",
   "metadata": {},
   "source": [
    "我们先将之前线性拟合的代码拷过来："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2033876f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:49:19.830225Z",
     "start_time": "2022-11-12T08:49:19.796724Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] ME(18928:18596,MainProcess):2022-11-12-16:49:19.810.600 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import mindspore.nn as nn\n",
    "import mindspore as ms\n",
    "from mindspore import dataset as ds\n",
    "from mindspore.common.initializer import Normal\n",
    "from mindvision.engine.callback import LossMonitor\n",
    "\n",
    "def get_data(num, w=2.0, b=3.0):\n",
    "    \"\"\"生成数据及对应标签\"\"\"\n",
    "    for _ in range(num):\n",
    "        x = np.random.uniform(-10.0, 10.0)\n",
    "        noise = np.random.normal(0, 1)\n",
    "        y = x * w + b + noise\n",
    "        yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32)\n",
    "\n",
    "def create_dataset(num_data, batch_size=16):\n",
    "    \"\"\"加载数据集\"\"\"\n",
    "    dataset = ds.GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label'])\n",
    "    dataset = dataset.batch(batch_size)\n",
    "    return dataset\n",
    "\n",
    "class LinearNet(nn.Cell):\n",
    "    \"\"\"定义线性回归网络\"\"\"\n",
    "    def __init__(self):\n",
    "        super(LinearNet, self).__init__()\n",
    "        self.fc = nn.Dense(1, 1, Normal(0.02), Normal(0.02))\n",
    "\n",
    "    def construct(self, x):\n",
    "        return self.fc(x)\n",
    "\n",
    "loss_fn = nn.L1Loss()\n",
    "dataset = create_dataset(num_data=160)\n",
    "net = LinearNet()\n",
    "optm = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4ce6995",
   "metadata": {},
   "source": [
    "初始化Model接口"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "456a814d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:50:52.126283Z",
     "start_time": "2022-11-12T08:50:52.108937Z"
    }
   },
   "outputs": [],
   "source": [
    "model = ms.Model(net, loss_fn, optm, metrics={\"MAE\": nn.MAE()})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f570f26",
   "metadata": {},
   "source": [
    "执行训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ea2e2cd5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:51:38.887475Z",
     "start_time": "2022-11-12T08:51:38.484068Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch:[  0/  1], step:[    1/   10], loss:[11.268/11.268], time:325.055 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    2/   10], loss:[9.084/10.176], time:0.969 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    3/   10], loss:[7.728/9.360], time:2.965 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    4/   10], loss:[8.207/9.072], time:1.006 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    5/   10], loss:[8.562/8.970], time:0.975 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    6/   10], loss:[6.964/8.635], time:0.983 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    7/   10], loss:[8.171/8.569], time:1.000 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    8/   10], loss:[7.971/8.494], time:1.013 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    9/   10], loss:[6.724/8.298], time:1.980 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[   10/   10], loss:[7.249/8.193], time:1.979 ms, lr:0.00500\n",
      "Epoch time: 362.098 ms, per step time: 36.210 ms, avg loss: 8.193\n"
     ]
    }
   ],
   "source": [
    "model.train(epoch=1, train_dataset=dataset, callbacks=LossMonitor(0.005))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e8b8012",
   "metadata": {},
   "source": [
    "模型评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "eaf62ea8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:52:33.907551Z",
     "start_time": "2022-11-12T08:52:33.839644Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] ME(18928:18596,MainProcess):2022-11-12-16:52:33.845.654 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc: {'MAE': 5.306867074966431}\n"
     ]
    }
   ],
   "source": [
    "eval_dataset = create_dataset(num_data=160)\n",
    "output = model.eval(eval_dataset)\n",
    "print(\"acc:\", output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "433fb311",
   "metadata": {},
   "source": [
    "## 使用自定义Metrics\n",
    "只需在Model指定metrics参数时，将字典的`value`指定为`自定义的Metrics`即可。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9b8946a",
   "metadata": {},
   "source": [
    "初始化Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "66d11c37",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:54:27.272315Z",
     "start_time": "2022-11-12T08:54:27.245145Z"
    }
   },
   "outputs": [],
   "source": [
    "model = ms.Model(net, loss_fn, optm, metrics={\"MAE\": MyMAE()})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9878a79",
   "metadata": {},
   "source": [
    "执行训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6b432dc7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:55:31.122800Z",
     "start_time": "2022-11-12T08:55:30.942452Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] ME(18928:18596,MainProcess):2022-11-12-16:55:30.949.455 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch:[  0/  1], step:[    1/   10], loss:[6.750/6.750], time:122.583 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    2/   10], loss:[6.009/6.380], time:0.957 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    3/   10], loss:[4.749/5.836], time:2.977 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    4/   10], loss:[4.023/5.383], time:0.999 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    5/   10], loss:[3.833/5.073], time:1.009 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    6/   10], loss:[2.682/4.674], time:1.002 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    7/   10], loss:[2.860/4.415], time:0.000 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    8/   10], loss:[2.852/4.220], time:2.961 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[    9/   10], loss:[3.126/4.098], time:0.960 ms, lr:0.00500\n",
      "Epoch:[  0/  1], step:[   10/   10], loss:[3.466/4.035], time:2.000 ms, lr:0.00500\n",
      "Epoch time: 159.111 ms, per step time: 15.911 ms, avg loss: 4.035\n"
     ]
    }
   ],
   "source": [
    "ds_dataset = create_dataset(num_data=160)\n",
    "model.train(epoch=1, train_dataset=ds_dataset, callbacks=LossMonitor(0.005))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a83b0acb",
   "metadata": {},
   "source": [
    "模型评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "f7b91555",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-11-12T08:55:56.224555Z",
     "start_time": "2022-11-12T08:55:56.147185Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] ME(18928:18596,MainProcess):2022-11-12-16:55:56.151.633 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'MAE': 3.333993172645569}\n"
     ]
    }
   ],
   "source": [
    "ds_eval = create_dataset(num_data=160)\n",
    "output = model.eval(ds_eval)\n",
    "print(output)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mindspore",
   "language": "python",
   "name": "mindvision"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
